Instrument guidance system for sinus surgery

ABSTRACT

A method for generating an augmented reality image for supporting the adjustment of the position of a surgical instrument during sinus surgery comprising the steps of: selecting at least one of sinus cells, cavities and orifices of the sinus in a pre-operative image by marking them with at least one geometric primitive or tube shaped object; using real-time intra-operative imaging for displaying the operating field; determining the relative positions at least between the real-time imaging of the patient and the pre-operative image data; computing a virtual image corresponding to a real-time image comprising at least one marked geometric primitive or tube shaped object based on the determined relative positions; combining the virtual image and the real-time intraoperative image for visualization in an augmented reality image.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of PCT/EP2015/075158 (WO2016/066759) filed on Oct. 29, 2015 claiming priority of UK PatentApplication No. GB 1501157.0 filed on Jan. 23, 2015 and U.S. provisionalapplication U.S. 62/073,512 filed on Oct. 31, 2014. The aforementionedapplications are hereby incorporated by reference in its entirety.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to a method for generating an augmentedreality image for supporting the adjustment of the position of asurgical instrument during sinus surgery

Brief Description of the Related Art

There are several diseases known which can cause airway narrowing orstenosis. Inflammatory conditions belong to those known diseases. Whilethere are several methods available to improve airway narrowing, balloondilatation may provide a gentle, uniform dilation with less airwaytrauma than endoscopic sinus surgery with cutting instruments.

Physicians use during surgical planning a simplified planning scheme,wherein the cells or cavities of the nose are painted on paper toachieve better orientation in the complex anatomy of the sinuses. Insuch planning schemes simple three-dimensional geometric primitives areused to define the position and size of nasal cavities. Such a schemesupports the physician to transfer acquired generalized medicalknowledge to the current situation in a patient and to detect therelevant anatomical structures and classify their geometric position.Such acute knowledge about the patient at hand is crucial to performsurgical interventions to achieve results within the medical state ofthe art.

In the literature, there are approaches to support the clinician byautomatic and semi-automatic segmentation methods of the nasal anatomyduring surgical planning. Here, the aim is to detect and identify thecavities of the nose in order to find or define the optimal access pathto the surgical region of interest (ROI).

Zein and colleagues described 3-D region growing segmentation (Zein etal., 2005 Bildverarbeitung für die Medizin 2005 (pp. 93-97). Thedescription discloses a contrast enhancement of 3-D image data andedge-preserving smooth filtering, a 3D region growing starting at seedvoxels, which fulfil intensity threshold criterion followed by apost-processing of segmentation to reject areas not belonging to thenasal system by detecting leakage regions. The results are used for a 3Dview of air space of paranasal sinuses and virtual endoscopy

Disadvantages related to the method disclosed by Zein et al are that themethod only works with air-filled nasal cells. Further, it does notprovide a differentiation and subdivision of individual nasal cells andthe detection of natural drainage pathways of the sinuses.

Lo and de Bruijne described a voxel classification based airway treesegmentation (Lo, P., & de Bruijne, M. 2008, Proc. SPIE). They describeda 3D region growing starting with main bronchus as seed point. Theydefine with K^(th) nearest neighbor (KNN) a local criterion using thefollowing image descriptors instead of using only image gray value:convolution with Gaussian, 1st and 2nd order derivates, gradientmagnitude, Eigenvalues of Hessian, Laplacian, Gaussian curvature, Eigenmagnitude, ratios of eigenvalues.

It is disadvantageous that the method disclosed by Lo and de Bruijne isonly applicable for segments air-filled cavities and adjusted for use tosegment airways in the lung and not suitable for the determination ofnatural drainage pathways of the sinuses.

Tingelhoff and colleagues disclose a comparison between the manual andsemi-automatic segmentation of nasal cavity and paranasal sinuses fromCT images (Tingelhoff et al., 2007, Proceedings of the 29th AnnualInternational Conference of the IEEE EMBS (pp. 5505-5508). This documentdiscloses a 3D region growing segmentation using AMIRA 4.1 software(Mercury Inc., now: Visage Imaging Inc.). It is disadvantageous thatonly segmentation of connected cavities is possible and a manualdefinition of seed points for region growing algorithm is necessary.

Moral and colleagues disclose a 3D region growing segmentation usingAMIRA 4.1 software for planning of a path from nostrils to maxillarysinuses, sphenoidal sinuses, ethoidal sinuses and frontal sinuses (Moralet al. 2007 Proceedings of the 29th Annual International Conference ofthe IEEE EMBS, pp. 4683-4686). The disadvantages are same as mentionedabove for the publication of Tingelhoff et al., namely that onlysegmentation of connected cavities is possible and a manual definitionof seed points for region growing algorithm is necessary.

WO 2013/012492 A2 discloses a method for displaying a paranasal sinusregion of a patient, comprising acquiring volume image data of theparanasal sinus region of the patient, identifying one or more airwayswithin the paranasal sinus region from the volume image data, displayingthe at least one or more airways and highlighting one or more portionsof the displayed one or more airways that are constricted below apredetermined value.

WO 2013/012492 A2 relates to CT/Cone-Beam imaging, adaptive segmentationbased on the disclosure of Pappas (Pappas, 1992, IEEE Transactions onSignal Processing, 40(4)), K-means classification (clustering) ofsegmented region based on their mean gray values (e.g. 4 clusters),voxel assignment to a cluster, external air removal using morphologicaloperators on tissue maps/clusters and user interaction for modificationof the result of the automatic segmentation, as can be taken from FIG.4. This figure shows an added user input step 405 allowing the user toprovide input that improves automatic segmentation, including edits tothe class map, for example. In step 405, the user further providesinstructions that modify the display of anatomy of the nasal region insome way. The modifications can be indicated interactively by viewerinstructions entered with reference to a displayed rendering of CTslices in a coronal, axial, sagittal, or other view. The user inputinstructions can be entered using a pointer device, such as a computermouse or joystick, for example, or using a touch screen as input device.Alternatively, the user may interact with the system using a 3Drendering of the nasal region. For example, in step 405 the user mayenter instructions that indicate that an ostium of the left maxillarysinus is blocked. The indication that a ostium is blocked will cause aspecific colour to display that cells may change. It is possible thatthe user removes sinus cells, bone and other tissue from the display, toskeletonize airways (compute medial lines within objects), computecross-sectional views along the skeletons (perpendicular to path) andhighlight important locations (e.g. locations of global or local minimumcross-sectional area that may occur at sinus ostia or locations at whicha drainage path is restricted, or points with high curvature).

The method disclosed in WO 2013/012492 A2 allows a virtual endoscopicview, provides a path finding algorithm and the registration ofsegmentation to a labelled atlas based on statistical data so that theanatomy is identified. Alternatively manual anatomy identification orlabelling by the user (using lists, audible prompt) is possible. Anatlas with irregularities helps to identify different anatomicalvariations (e.g. agger nasi cell). Cell properties can be displayed(volume (natural and air), location, presence of polyps or infections)and simplified graphical representation of anatomical structures basedon segmentation results can be generated. This document discloses asystem to execute segmentation and display of paranasal cavities

A disadvantage of the method disclosed in WO 2013/012492 A2 is thedifficulty for the user to prove the results of automatic segmentationand optimize it, especially to separate connected cells. The method doesnot allow to separate connected cells, a corresponding description ismissing. In addition, there is no description how to segment blockedostia.

WO 2009/120196A1 discloses a method of and a system for a 3D workstationfor security and medical applications. A rendering method of volumetricdata is described, including highlighting detected regions using thecontour of the object on 2D displays and 3D stereoscopic displays. Thecontour colours are differently from rendering the volumetric datawithout highlighting. This document describes only the highlighting of amarked region in 2-D or 3-D views, but no description how the regionsare detected is provided.

EP 1941449 B1 discloses a method for rendering a surface indicative ofan interior of a colon. The method comprises using volumetric data (202)indicative of the interior anatomy of a human colon to render a surface(102) indicative of an interior of the colon. The method ischaracterized by the steps of identifying a region of the renderedsurface, which is suspected to include residual stool, and highlightingthe identified region (104) on an image of the rendered surface (102).The contour of the region of interest (ROI) is highlighted in virtualendoscopic view. This document discloses only a rendering method relatedto the colon.

WO 2008/021702 A2 discloses a method of quantifying a sinus condition ofat least one sinus cavity of a patient. The method comprises the stepsof generating an image of the patient, locating the at least one sinuscavity in the image and quantifying the sinus condition of the at leastone sinus cavity based on the image, the automatic density-baseddetection and location of sinus cavities, a manual identification ofsinus cavities by a technician and quantification of the amount of fluidor polys in a sinus cavity to determine the sinus condition or progressof sinus condition. WO 2008/021702 A2 does not provide theimplementation of the manual or automatic segmentation of sinus cavities

In summary, the existing planning methods and systems provide someapproaches for automatic segmentation of mainly air-filled sinusescavities. The automatic segmentation methods, e.g. 3D region growing or3D adaptive segmentation with k-means clustering, work well withair-filled sinus cavities. The challenge in the planning of sinussurgery, however, lies rather in a separation and identification ofindividual cells in particular under the absence of air.

In case of diseases such as inflammation of the sinuses or polyps,single or multiple cells are filled with tissue or fluid and theautomatic segmentation methods are likely to fail due to marginal greyvalue differences of some cartilaginous cell walls and mucosa orpathological cell filling. Also, the identification and labelling ofseparated cavities is an unsolved problem especially in the case ofpathological anatomy. Also the different quality of the 3D image data isoften a problem for automatic methods, which require a high resolutionof the volume image data and a normalized, or uniform grey scale densitymap in order to guarantee satisfying results. Especially image data fromolder devices and Cone beam computed tomography (CBCT) often do not meetthese conditions.

Systems known from the state of the art do not provide support for aplanning scheme based on patient-specific 3D image data of the humanbody, particularly of the paranasal sinuses. Therefore, up to date theplanning scheme can only be manually performed on paper with thefollowing disadvantages:

-   -   Spatial incorrectness    -   Error-prone scaling/size assignment    -   Used cuboids or cylinders describe the shape of the cells        sometimes inadequately    -   Insufficient applicability to the surgical intervention

Thus, there is a need for a computer-assisted method for fast andeasy-to-use segmentation of the relevant nasal cells during the surgicalplanning according to simplified planning schemes. The results of theplanning are intended to be in such a format that they also can be usedand visualized intra-operatively by surgical assistance technology suchas a surgical navigation system. For endoscopic or microscopic surgery,navigation systems with augmented-reality support allow the display ofplanning data superimposed in real-time endoscopic or microscopic cameraimages.

The perception of depth information of superimposed data in theaugmented reality representation is usually also a problem, since thedistance between the visualised object and the camera is often difficultto recognize. Another challenge is the intraoperative positioning andalignment of instruments relative to planning object, particularly usingthe real-time endoscopic or microscopic imaging.

SUMMARY OF THE INVENTION

The present invention provides a method for generating an augmentedreality image for supporting the adjustment of the position of asurgical instrument during sinus surgery comprising the steps of:selecting at least one of sinus cells, cavities and orifices of thesinus in a pre-operative image by marking them with at least onegeometric primitive or tube shaped object; using real-timeintra-operative imaging for displaying the operating field; determiningthe relative positions at least between the real-time imaging of thepatient and the pre-operative image data; computing a virtual imagecorresponding to a real-time image comprising at least one markedgeometric primitive or tube shaped object based on the determinedrelative positions; combining the virtual image and the real-timeintraoperative image for visualization in an augmented reality image. Itis intended to display the generated augmented reality image to thesurgeon or user.

The method may further comprise that the virtual image additionallycomprises visualisation elements for an enhanced 3D perception of thegeometric primitive or tube shaped object.

One configuration of the visualisation elements is a periodicallytexturing of the surface of the tube shaped objects along their central3D axes, wherein the texture of the tube shaped object may comprise astriped pattern.

Another configuration of the visualisation elements is the use of asequence of at least two circles or polygons to mark selected orificesof the sinus in real-time intraoperative images. The sequence of atleast two circles or polygons may surround the tube shaped objectsmarking the selected orifices in the sinus.

The centre of the at least two circles or polygons may furthercorrespond to the centre of the tube shaped object and the at least twocircles or polygons of a sequence of at least three circles or polygonsmay be arranged in a constant distance. The same applies for the use ofat least three circles or polygons of a sequence of at least threecircles or polygons that are arranged in a constant distance to itsdirect neighbours along the tube shaped object.

The relative position between the surgical instrument, whose positionadjustment is to be supported, and the real-time imaging may bedetermined and the display of the visualisation elements in theaugmented-reality image can be adjusted to the spatial position ororientation of the navigated surgical instrument.

The progress of adjusting the position of the surgical instrumenttowards the end point of the tube shaped object may be visualized byhiding visualisation elements between the starting point of the tubeshaped object and the position of the instrument or by modifyingcolours, texture, transparency, thickness, shape or intensity of thevisualisation elements.

The deviation of the spatial position or orientation of the instrumentfrom the tube shaped objects may be visualized by modifying, adding,hiding or highlighting visualisation elements or parts of thereofdepending on the necessary support for adjusting the position of theinstrument.

It is further intended that the real-time intra-operative image may beobtained from optical systems like endoscopes or operating microscopes.

The determination of the relative position of the camera of the realtime imaging may be based on using an optical tracking system orelectro-magnetic tracking system or a combination thereof or that isbased on the continuous analysis of the real-time intra-operative imagedata.

It is envisaged to use pre-operatively three-dimensional image data likeCT, MRI, Cone Beam CT or time dependent image data.

Further, the surgical instrument, whose position adjustment is to besupported, may be selected from the group of endoscopes, cameras,scalpels, catheters and balloon catheters or combinations thereof andthe surgical instrument whose position adjustment is to be supported maybe flexible. The leading part of the surgical instrument whose positionadjustment is to be supported may furthermore be inflatable.

The sinus cells, cavities and orifices of the sinuses may be selectedmanually, semiautomatic or automatically.

It is envisaged that the selection of the sinus cells, cavities andorifices of the sinuses may comprise manual pre-segmentation by definingenclosing geometric primitives in the pre-operative 3D image forgenerating initial cell envelopes or body cavities, analysing theanatomy of the sinus cells, cavities and orifices of the sinuses withinthe pre-segmented geometric primitives, using the result of the analysisfor superimposing geometric primitives of at least one of the sinuscells, cavities and orifices of the sinuses in the real-timeintra-operative image.

It is further intended that selecting at least one of sinus cells,cavities and orifices of the sinus in a pre-operative image is achievedby marking them with geometric primitives or tube shaped objectscomprises the use of selections marked in previous pre-operative imagedata.

The geometric primitive may be selected from a group comprising acuboid, sphere, ovoid, cylinder and ellipse.

The use of the above described method for generating an augmentedreality image for supporting the adjustment of the position of asurgical instrument during sinus surgery is another object of thepresent disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be described by figures and examples. It isobvious for a person ordinary skilled in the art that the scope of theinvention is not limited to the disclosed embodiments. It shows:

FIG. 1 is a pre-operative real-time image superimposing geometricprimitives and the natural drainage pathways

FIG. 2 is an intra-operative real-time image superimposing geometricprimitives and the natural drainage pathways

FIG. 3 is an intra-operative real-time image superimposing geometricprimitives and the natural drainage pathways after opening a cavity ofthe sinuses

FIG. 4 is an intra-operative real-time endoscopic image withsuperimposed tube shaped objects surrounded by circles

DETAILED DESCRIPTION OF THE INVENTION

The present invention provides a fast, easy-to-use and intuitive methodfor generating an augmented reality image for supporting the adjustmentof the position of a surgical instrument during sinus surgery. Further,the present invention provides a method for supporting computer assistednavigation and opening or reopening natural cavities and orifices of thehuman body, wherein the treatment of the human or animal body is notpart of the instant invention. The method is based on a manual orautomatic selection or segmentation of sinus cells, cavities and/ororifices of the sinuses. The method is suitable for computer assistedlabelling of cavities like sinus cavities for instance in the 3-Dpatient image data according to the planning scheme described in theliterature and taught in the training of surgeons.

It is an advantage of the subject matter of the instant invention thatit is for the first time possible to assist a surgeon during sinussurgery by superimposing geometric primitives or the possible andpredicted course of the natural drainage pathway with tube shapedobjects and visualisation elements to support the adjustment of theposition of the surgical instruments. A start or starting point likewisean end point refer to the beginning or end of a tube shaped object. Itis intended that the end point is located at the targeted orifice of thesinus.

An input device within the meaning of the present invention comprisesany peripheral equipment used to provide data and control signals to aninformation processing system such as a computer or other informationappliance. A computer mouse or a touch-screen are examples of an inputdevice according to the present invention.

Using the manual segmentation, the invention provides a method beingcapable of performing an automated improvement and analysis of themanual segmentation leading to a surprisingly better adjustment of thecell envelope, identification of natural drainage pathways and generatesan improved input for the automatic optimization of the cavityvisualization based on the 3-D image data of the patient.

The invention describes a method for the identification andvisualization of natural cavities in a microscope image for performingparanasal sinus balloon dilatation.

The procedure of computer assisted segmentation of sinus cells requiresthe manual pre-segmentation of cells by defining enclosing geometricprimitives and can further comprise the following steps:

-   -   a. Identification of cells based on manual input or automatic        detection,    -   b. Analysis of the contents of the segmented areas and the        connecting space based on the 3-D image data of the patient,    -   c. Adjustment of the properties,    -   d. Providing an overlay image showing the geometric primitives        as well as the position of the determined natural cavities or        openings.

The process for segmenting a cell is as follows:

In a first step, manual or automatic selection of the type of thegeometric primitive for manual marking of cells or cavitiesis done. Ifapplicable, only one method can be offered or pre-selected. The user mayidentify the cell that should be segmented from a list. Depending on thecell type, a type of geometric primitive is pre-selected but can bechanged by the user.

If the identification of an anatomical cell or natural cavity is notperformed in this step, it may be subsequently assigned. The input orassignment of a label can be done by textual input using akeyboard/screen keyboard or by voice. Alternatively, predefined labelsselected from a list and a selected cell to be assigned. The colour canbe chosen for example from a group of predefined colours or be set in aspecial colour selection dialog box in which any colour can be selected.

In a second step, a slice image of the 3-D image data is selected, whichshows the cell or natural cavity to be selected. The orientation of theslice image can be chosen, preferably axial, coronal or sagittal, but afree orientation is also possible.

The following step of an initial definition of the geometric shape byuser interaction in the selected slice image, comprises the steps of:

-   -   a. User interaction in case of a cuboid: Define the cell in the        slice view by clicking opposite vertices or “raising” of the        rectangle. The sequence of user interaction using an input        device such as a computer mouse can be in case of “raising” of a        rectangle as follows:        -   i. Pressing the computer mouse button at any position (x1,            y1) in the selected slice image        -   ii. Moving the computer mouse with a pressed button to a            second position (x2, y2) in the selected slice image        -   iii. Releasing the computer mouse button    -   When the rectangle is defined for the first time, a depth for        the cuboid must be assigned. In the simplest case, a depth of 0        can be set. Alternatively, the depth d of the cuboid can be        calculated by evaluating the initial height h and width w of the        rectangle in the average image with w=(x2−x1) and h=(y2−y1):

$d = \sqrt{\frac{w^{2} + h^{2}}{2}}$ or $d = \frac{{w} + {h}}{2}$

-   -   The box is positioned relative to the slice image such that that        the selected slice image bisects the cuboid in depth.    -   b. User interaction in case of a sphere: Using an input device        by pressing the computer mouse button at the position of the        sphere centre and then “raising” the sphere. The sequence of        user interaction can be implemented as follows:        -   i. Pressing the computer mouse button at any position (xc,            yc) in the selected slice image        -   ii. Moving the computer mouse to a second position (x2, y2)            in the selected slice image        -   iii. Releasing the computer mouse button    -   The position and size of the sphere is obtained by the centre        (xc, yc) and the radius r of the sphere with

r=√{square root over ((xc−x2)²+(yc−y2)²)}

-   -   c. User interaction in case of a ellipsoid: The definition of        the initial position and size of the ellipsoid can be done in        two ways:        -   i. In analogy to cuboid definition (e.g. raising from corner            to corner), except that an ellipse instead of a rectangle is            visualized in the slice image. The ellipse is obtained from            the two corner points (x1, y1) and (x2, y2) as follows:            -   Centre of the ellipse corresponds to the centre of the                rectangle            -   Vertices of the ellipse are located on the rectangle                boundary            -   Direction of the major axis of the ellipse corresponds                to the direction of the longer rectangle side        -   ii. In analogy to sphere definition: Clicking on slice image            to define the centre point and raise the ellipse by moving            the computer mouse while keeping the mouse button pressed.            The size of the ellipse is chosen so that the major and            minor axes of the ellipse are aligned horizontally or            vertically to the slice image and that the collected            positions are located within the ellipse.    -   The calculation of the depth of the ellipsoid can be made        equivalent to the depth calculation of the box.    -   d. User interaction in case of a cylinder: The definition of the        base of the cylinder can be done in the selected slice image.        User interaction can be carried out in analogy to the definition        of a sphere. The initial height h of the cylinder can, for        example, dependent are determined by the radius r of the defined        surface area. Here as advantageous to determining h=2*r has been        established.

In the final step of segmenting a cell or natural cavity, the adjustmentof the position and size of geometric primitives is carried out byshifting corner points or centre points or edges in any slice images ofthe 3D image data carry out size of geometric primitives. Here, thesectional contour of the currently defined geometry and the currentlyselected image slice is calculated in each case.

Clicking and dragging the section contour or the area inside the contourcan change the size or the position of the geometric primitives. In thecase of cuboids, clicking and dragging vertices or edges of therectangle, which is visualized in the current slice image, can changethe size and position of the cuboid. In the case of the availability ofa multi-touch screen, gestures can be used to change the geometricprimitives through simultaneous use of multiple fingers.

The identification and computer assisted labelling of the segmentedcells or natural cavities can be helpful during the planning processand/or the surgical intervention when the segmented cell or naturalcavity will be visualized in different views. The followingpossibilities may be used to identify the cells or natural cavities (butnot limited to):

-   -   Selecting predefined labels prior to segmentation based on        anatomical list.    -   Selecting predefined labels or entering custom labels during or        after segmentation    -   Automatic assignment of predefined labels based on anatomical        Atlas    -   Automatic assignment of predefined labels based on the order of        segmentation    -   Automatic assignment of predefined labels based on the spatial        relation of the objects to each other.    -   Some implementations of the invention may not require the        identification to derive sufficient quality results

Based on the manually segmented and potentially identified geometricprimitives, the data values of the 3-D image data are analysed insidethe selected geometric primitive and its vicinity with the aim tocalculate the correct anatomical shape of the cell and derive additionalrelevant properties such as (but not limited to): filling, diseasestatus, volume, drainage pathway, entry point or exit point. For each3-D image element (voxel) it has to be decided whether the voxel belongsa.) completely to the cell, b.) not at all to the cell, c.) partially tothe cell with a given subdivision boundary. The differentiation of thecell interior from the rest within the area defined by geometricprimitive and its surrounding can be implemented as follows (but notlimited to):

-   -   The assignment bases only on the data value of the 3-D image        element, the determination of the threshold can be selected        automatically e.g. by the method of Otsu (Otsu, N., 1979, IEEE        Transactions on Systems, Man and Cybernetics, 9(1), 62-66) or        manually by the user    -   3-D region growing with one or more seed points which are        automatically or manually selected    -   Adaptive segmentation methods such as (Pappas, 1992,        Transactions on Signal Processing, 40(4))    -   Analysis of the gradients in a 3-D data set starting from the        centre of the geometric primitive.

Based on the results of the analysis of the contents of manuallysegmented geometric primitives, the segmentation and thus thevisualization of the cell envelope can be adjusted. The implementationcan be realized in one of the following ways:

-   -   a. The envelope is calculated based on the automatically        segmented cell anatomy. The cell envelope can be generated for        example as a triangle mesh which encloses the voxels        corresponding to the cell using the marching cubes method        (Lorensen & Cline, 1987, Proceedings of the 14th annual        conference on Computer graphics and interactive techniques (pp.        163-169). FIGS. 6 and 8 show the adjusted segmentation of the        cell according to the anatomy of the patient within a previously        manually defined cuboid in a slice image and a 3D view.    -   b. Adjustment of the manually defined geometric shape based on        the results of the evaluation of the 3D image data. Here one can        proceed as follows:    -   i. Selection of the type of geometric primitive may        automatically based on the initial segmentation, the cell        identification, the associated label, user input or the        anatomical shape of the cell.    -   ii. Optimization of the position, orientation and size of the        geometric primitive such that the geometric shape describes as        good as possible the correct anatomical shape of the cell. In        case of a sphere, one solution can be to determine the smallest        enclosing sphere which encloses all voxels corresponding to the        cell. The determination of the position and size of the sphere        uses optimization methods which minimizes the radius under the        constraint

(c _(x) −x _(i))²+(c _(y) −y _(i))²+(c _(z) −z _(i))² ≦r ²,

-   -   where c_(x), c_(y), c_(z) denote the coordinates the sphere        centre point, and x_(i), y_(i), z_(i) denotes the centre point        of the i-th voxel which is a part of the anatomical cell        segmentation.    -   Another approach to the determination of the sphere is the        positioning the sphere in the centre of all voxels belonging to        the cell. The radius of the sphere can then be computed such        that the volume of the anatomical cell is equivalent to the        volume of the sphere.    -   Both of these approaches, the optimization of the ball as the        complete envelope of the cell or as a sphere having the same        volume as the cell, can be implemented as well for other        geometric primitives. In these cases further constraints can be        defined e.g. on the orientation of the primitives, so that only        the position and size are allowed to be changed in comparison to        the result of the manual pre-segmentation step or the        orientation of the geometric primitive corresponds to a given        reference coordinate system.

In addition to the shape, position and size of a cell or natural cavity,e.g. in the sinuses, the location of the ostium is an importantinformation for the surgeon during the planning and performing asurgical intervention.

The segmentation of the natural drainage pathway of a nasal cell can beperformed by manual marking multiple points in the slice images of the3D image data, which can be connected according to the order of markingor according to their location.

Alternatively, the results of the performed cell segmentation can beused to automatically determine the location of the natural drainagepathway. For this purpose, it is possible to determine either theoptimal or shortest path to adjacent already segmented cells. Here,well-known pathfinding algorithms may be used which interpret the 3Dimage data as a graph. The gray values or gray value changes in the 3-Dimage data are used to calculate the cost between the nodes of thegraph.

If in addition the main nasal cavity with connected sinuses isautomatically segmented e.g. by means of 3D region growing, the naturaldrainage pathway of a segmented cell to the main nasal cavity can alsobe determined. Here, pathfinding algorithms, such as the A* algorithm,can be used, too.

The presentation of the natural drainage pathway of a cell can berealized in the slice images, in the 3-D view, in a virtual endoscopicview or in an endoscopic augmented reality view as either (but notlimited to):

-   -   linked line segments    -   splines    -   linked, possibly curved cylindrical segments    -   meshes    -   voxels

For surgical planning, it is often important for the user to knowparameters and properties of the segmented cell to perform the treatmentoptimally. These parameters can include the width, height, depth andvolume of the adjusted cell envelope, which can be displayed to theuser. In addition, it is possible to deduce from the gray values of the3D image data, whether the cell is filled with air or fluid or tissue.Hence, an automatic assignment to the health of the cell can be met.Also the shape and diameter of the natural drainage pathway can beanalysed based on the 3D image data in order to identify problem regionsin the anatomy of the patient and to highlight the visualization of thecell if necessary.

Based on the results of individual cell analysis, a global cell analysisand inter-space analysis can be conducted to adjust cell propertiesbased on individual and global analysis results. The following analysissteps are possible (not limited to):

-   -   Linkage between cells based on shortest path, connectivity,        anatomical aspects    -   identification of additional inter-space cells    -   size correction of cells    -   Calculation of recommended surgical pathway    -   identification of anatomical aberrations, pathologies    -   Identification of critical anatomical sites such as nerves,        blood vessels

Based on the identified properties additional visualization marks may beadded as follows (but not limited to):

-   -   Entry and exit locations into neighbouring cells may be        visualized    -   The pathways between entry and exit as trajectories    -   The disease status and filling with additional symbols and        colours    -   Volume size as textual output    -   Identification label as textual output    -   Addition of audiovisual warning labels and sound for alarm        regions

The visualisation in slice images comprises:

-   -   Sectional illustration of the pre-segmented geometric primitives        and/or the adjusted cell envelope in the plane of the slice        image. The display can show the boundaries of the geometric        primitive and/or the cell envelope (as shown in FIG. 5) or its        sectional area is marked opaque or semi-transparent (FIG. 6)    -   Display of labels for cell identification (this can be textual        information or numbers, symbols are also possible)    -   Highlighting the currently selected cell (e.g. by using a solid        line to show the currently selected cell while all cells are        displayed with dashed lines, alternatively the colour, line        width or opacity can be changed to highlight the currently        selected cell)    -   Display of the defined natural drainage pathways as a        intersection of the path with the plane of the slice image in        form of a point or highlighted area        -   The visualization in a 3D view comprises:    -   Perspective or orthographic representation of the geometric        primitives and/or the adjusted cell envelope in a 3D view of the        nasal cavities    -   Display of labels to identify the cells (this can be textual        information or numbers, symbols are also possible)    -   Display of the segmented natural drainage pathways as linked        line segments, splines or linked, possibly curved cylindrical        segments    -   By hiding the surface of the 3D image data, the relative        position of the segmented cells to one another can be conceived        well by user.

The virtual endoscopic view is comparable to the 3D view showing thesegmentation in the 3D image data, but it simulates the imaging of thesurgical field through endoscope optics. In comparison to the 3D view,the appearance of the surfaces of the 3D image data and the cameraperspective properties are adjusted to correspond to normal endoscopicimaging.

During the intervention, the imaging properties of the camera can alsouse the parameters of the camera model of the currently used endoscope,which were estimated during an intra-operative endoscope calibrationprocess. This enables the display of the real the corresponding virtualendoscopic image side by side. This allows the surgeon to transfermentally the visualised segmentation from the virtual to the realendoscopic image.

In the augmented reality view, the planning data are displayed as ansuperimposition on the images of a real camera system, such as anendoscope or operating microscope. For this purpose, it is necessary toknow accurately the imaging properties of the camera system to achievethe marking of the segmentation at the correct position in the realendoscopic image. Based on the known imaging properties of theintraoperative camera system and the continuously measured relativeposition between the real-time imaging system and the pre-operativeimage data, a virtual image corresponding to the current real image iscomputed comprising at least one marked geometric primitive or tubeshaped object.

The already mentioned difficulties in displaying depth information or a3D-aspect apply in particular to the superimposed presentation of tubeshaped objects in the intra-operative real-time camera image having asemi-transparent surface. It is therefore advantageous to additionallydisplay visualisation elements in order to enhance the 3D perception ofthe tube shaped object.

One implementation of such visualisation elements is texturing thesurface of the tube shaped objects or the display of the surface withrepeating patterns in the intra-operative real-time camera image, e.g.along the central axis of the tube shaped object. This results in asignificantly improved display of the 3D pathway of a tube shapedobject, especially along the direction corresponding to the line ofsight of the camera system.

A further advantageous implementation of visualisation elements forenhancing the presentation the 3D pathway of a tube shaped object, isthe additional display of simple geometric shapes along the 3D pathwayof the selected orifices. In particular, circles with constant distancesalong the 3D pathway provide a very good spatial perception. The centresof the circles should be placed on this path, the plane of the circleshould be perpendicular to the central 3D axis of the path. A similareffect can be achieved using other simple geometric shapes, which enablethe user to derive the alignment of the shapes relative to the cameraposition from its mapping in the intraoperative real-time camera image.These shapes include particularly polygons (esp. rectangles, squares),possibly with rounded corners or arrows in the direction of the centralaxis of the tube shaped object.

During surgery, the superimposed presentation of planning objects andthe visualisation elements for enhanced 3D perception can be modifiedcontinuously depending on spatial position and orientation of thenavigated instrument. Especially the modification of visualisationelements of a tube shaped object marking an orifice is advantageous fordisplaying the progress in reaching the end of the pathway or adeviation of the spatial position or orientation of the instrument tothe central axis of the planned tube shaped object.

The superimposed display of progress in reaching the end of the pathwaycan easily be realized by hiding visualisation elements between thestarting point of the pathway or tube shaped object and thecorresponding position of the current position instrument on thepathways. Also a modification of used colours, texture, transparency,thickness, shape or intensity of the visualisation elements isconceivable.

The superimposed display of the deviation of the spatial position ororientation of the instrument can be realized by adding, hiding orhighlighting visualisation elements or parts of them depending on theposition of the instrument. If circles are used as visualisationelements, circle segments in the direction towards the position of theinstrument can be additionally displayed or highlighted in order toindicate its relative position to the planned pathway.

During the surgical intervention, the results of the nasal cellsegmentation can also be used by surgical navigation systems to show thedistance between the tool centre point of the currently navigatedinstrument and the nearest point on the envelope or centre of thesegmented cells or the natural drainage pathway. The distance can bedisplayed in text in a separate user interaction widget or as a textualoverlay in the augmented reality view or the virtual endoscopic view.

The novelty of the invention is based on a manual pre-segmentation forthe coarse localization of the cells or natural drainage pathways in theslice images of the 3D image data of the patient in combination with animproved display of 3D aspects in intra-operative real-time cameraimage. The results of the pre-segmentation can be optimized regardingshape, position, orientation and size by evaluating the 3D image data ofthe patient. This allows a reliable and computer-aided display of 3Dinformation for surgery planning especially in cases of pathologicalanatomy of the paranasal sinuses. The intraoperative improvedaugmented-reality visualisation using additional visualisation elementsleads to a significantly better 3D perception of the position or shapeof planning data in the intraoperative real-time image data, alsoindicating the relative position of the currently navigated instrumentfor improved instrument guidance.

A further advantage of the method for generating an augmented realityimage for supporting the adjustment of the position of a surgicalinstrument during sinus surgery is the improved possibility to use themethod for supporting the navigation or guiding a surgicalinstrument—like a balloon catheter—through the determined drainagepathways or natural cavities although they are not visible in the realpicture. Producing an overlay image showing the geometric primitives aswell as the calculated natural cavities enables a surgeon to navigate aballoon catheter to the position of the natural sinus drainage pathwaysfor instance and to reopen the respective part of the drainage pathwayby inflating the balloon of the balloon catheter.

The advantages of the invention can be summarized as follows:

-   -   a. Easy user interaction for manual segmentation of cells and        natural cavities in 3D image data by geometric primitives    -   b. Automatic optimization of manual segmentation based on 3D        image data usually without any further user interaction    -   c. Automated detection of the natural drainage pathways on the        basis of previous cell segmentation and the 3D image data of the        patient    -   d. Intuitive and improved 3D visualization of segmentation or        natural cavities during planning and intervention    -   e. Support for the adjustment of the position of surgical        instruments like balloon catheter to the determined position of        natural cavities or drainage pathways    -   f. Enabling minimal invasive surgical interventions without the        need to see the cavities in a real image

FIG. 1 shows an image overlaying the geometric primitives (cubes) andthe calculated drainage pathways of the sinuses (tubular structures).

FIG. 2 shows an overlay image showing the geometric primitives andcorresponding natural drainage pathways of sinuses. One of the drainagepathways is open and the other one is closed. The method of the presentinvention enables to guide a balloon catheter to the still closednatural drainage pathway and open it by inflating the balloon of theballoon catheter.

FIG. 3 shows 3 an opened natural drainage pathway of the sinuses. Thesuperimposed position of the natural drainage pathway (tubularstructure) allows also for the further guidance of the surgicalinstrument like a balloon catheter through the drainage pathway.

FIG. 4 shows an intra-operative real-time image superimposingtube-shaped objects surrounded by circles. Although the diameter of thecircles has to be understood as being constant in 3D, the diameter ofthe circles appears to decrease in the course of the superimposedtube-shaped object from the foreground to the background of the 2Dimage.

What is claimed is:
 1. A method for generating an augmented realityimage for supporting the adjustment of the position of a surgicalinstrument during sinus surgery comprising the steps of: a. selecting atleast one of sinus cells, cavities and orifices of the sinus in apre-operative image by marking them with at least one geometricprimitive or tube shaped object, using real-time intra-operative imagingfor displaying the operating field, b. determining the relativepositions at least between the real-time imaging of the patient and thepre-operative image data, c. computing a virtual image corresponding toa real-time image comprising at least one marked geometric primitive ortube shaped object based on the determined relative positions, d.combining the virtual image and the real-time intraoperative image forvisualisation in an augmented reality image.
 2. The method of claim 1,wherein the virtual image additionally comprises visualisation elementsfor an enhanced 3D perception of the geometric primitive or tube shapedobject.
 3. The method of claim 2, wherein the visualisation elementscomprising a sequence of at least two circles or polygons are used tomark selected orifices of the sinus in real-time intraoperative images.4. The method of claim 3, wherein the sequence of at least two circlesor polygons surrounds the tube shaped objects marking the selectedorifices in the sinus.
 5. The method of claim 3, wherein the centre ofthe at least two circles or polygons corresponds to the centre of thetube shaped object.
 6. The method of claim 3, wherein the at least threecircles or polygons of a sequence of at least three circles or polygonsare arranged in a constant distance to its direct neighbours along thetube shaped object.
 7. The method of claim 2, wherein the visualisationelements comprise a periodically texturing of the surface of the tubeshaped objects along their central axes.
 8. The method of claim 7,wherein the texture of the tube shaped object comprises a stripedpattern.
 9. The method of claim 2, wherein the relative position betweenthe surgical instrument, whose position adjustment is to be supported,and the real-time imaging is determined and the display of thevisualisation elements in the augmented-reality image is adjusted to thespatial position or orientation of the surgical instrument.
 10. Themethod of claim 9, wherein the progress of adjusting the position of thesurgical instrument towards the end point of the tube shaped object isvisualized by hiding visualisation elements between the starting pointof the tube shaped object and the position of the instrument or bymodifying colours, texture, transparency, thickness, shape or intensityof the visualisation elements.
 11. The method of claim 9, wherein thedeviation of the spatial position or orientation of the instrument fromthe tube shaped objects is visualized by modifying, adding, hiding orhighlighting visualisation elements or parts of thereof depending on theposition of the instrument.
 12. The method of claim 1, furthercomprising the step of displaying the combined image.
 13. The method ofclaim 1, wherein the real-time intra-operative image is obtained fromoptical systems like endoscopes or operating microscopes.
 14. The methodof claim 1, wherein determining the relative positions is based on usingan optical tracking system or electromagnetic tracking system or acombination thereof or is based on the continuous analysis of thereal-time intra-operative image data.
 15. The method of claim 1, whereinpre-operative three-dimensional image data like CT, MRI, Cone Beam CT ortime dependent image data are used.
 16. The method of claim 1, whereinthe surgical instrument, whose position adjustment is to be supported,is selected from the group of endoscopes, cameras, suction tubes,probes, clamps, scalpels, catheters and balloon catheters orcombinations thereof.
 17. The method of claim 1, wherein the surgicalinstrument, whose position adjustment is to be supported, is flexible.18. The method of claim 1, wherein the leading part of the surgicalinstrument, whose position adjustment is to be supported, is inflatable.19. The method of claim 1, wherein the sinus cells, cavities andorifices of the sinuses are selected manually, semiautomatic orautomatically.
 20. The method of claim 1, wherein the selection of thesinus cells, cavities and orifices of the sinuses comprises a. manualpre-segmentation by defining enclosing geometric primitives in thepre-operative 3D image for generating initial cell envelopes or bodycavities; b. analysing the anatomy of the sinus cells, cavities andorifices of the sinuses within the pre-segmented geometric primitives;and c. using the result of the analysis for superimposing geometricprimitives of at least one of the sinus cells, cavities and orifices ofthe sinuses in the real-time intra-operative image.
 21. The method ofclaim 1, wherein selecting at least one of sinus cells, cavities andorifices of the sinus in a pre-operative image by marking them withgeometric primitives or tube shaped objects comprises the use ofselections marked in previous pre-operative image data.
 22. The methodof claim 1, wherein the geometric primitive is selected from a groupcomprising a cuboid, sphere, ovoid, cylinder and ellipse.